Machine learning systems and methods for improved localization of image forgery

ABSTRACT

A system for improved localization of image forgery. The system generates a variational information bottleneck objective function and works with input image patches to implement an encoder-decoder architecture. The encoder-decoder architecture controls an information flow between the input image patches and a representation layer. The system utilizes information bottleneck to learn useful residual noise patterns and ignore semantic content present in each input image patch. The system trains a neural network to learn a representation indicative of a statistical fingerprint of a source camera model from each input image patch while excluding semantic content thereof. The system can determine a splicing manipulation localization by the trained neural network.

RELATED APPLICATIONS

The present application claims the benefit of U.S. ProvisionalApplication Ser. No. 62/865,414 filed on Jun. 24, 2019, the entiredisclosure of which is expressly incorporated herein by reference.

BACKGROUND Technical Field

The present disclosure relates generally to the field of machinelearning. Specifically, the present disclosure relates to machinelearning systems and methods for improved localization of image forgery.

RELATED ART

Photo-realistically altering the contents of digital images and videosis problematic as society becomes increasingly reliant on digital imagesand videos as dependable sources of information. Altering image contentsis facilitated by the availability of image editing software andaggravated by recent advances in deep generative models such asgenerative adversarial networks (GAN). Digital image forensics focuseson this issue by addressing critical problems such as establishing theveracity of a digital image (e.g., manipulation detection), localizing atampered region within the image (e.g., manipulation localization), andidentifying an alteration type. Different alteration types requiredifferent forensic techniques. One type of alteration includesintroducing foreign material into an image. For example, splicing can beutilized to insert a part of one image into another image or inpaintingcan be utilized to insert an object into an image via a specializedalgorithm. Semantic information has had limited success in solving suchoperations because skilled attackers utilize semantic structures to hideimage alterations. Non-semantic pixel-level statistics have proven moresuccessful since these statistics amplify low-level camera modelspecific distortions and noise patterns (i.e., a camera model's digitalfingerprint). A camera model digital fingerprint can aid in resolving anintegrity of an image by determining whether the camera modelfingerprint is consistent across an entirety of the image. Severalhand-engineered, low-level statistical approaches have been explored.However, given the aforementioned availability of image editing softwareand the technological improvement of recent deep generative models,there is a need for forensic algorithms that can provide data-drivendeep learning solutions for the localization of image forgery.

Therefore, there is a need for machine learning systems and methodswhich can improve the localization of image forgery while improving anability of computer systems to more efficiently process data. These andother needs are addressed by the machine learning systems and methods ofthe present disclosure.

SUMMARY

The present disclosure relates to machine learning systems and methodsfor improved localization of image forgery. The system generates avariational information bottleneck objective function and works withinput image patches to implement an encoder-decoder architecture. Theencoder-decoder architecture controls information flow between the inputimage patches and a representation layer. The system utilizesinformation bottleneck to learn useful noise-residual patterns anddiscard semantic content present in each input image patch. Inparticular, the system extracts noise-residual patterns by consideringlearned local noise models and learns a suitable representation (e.g., astatistical fingerprint of a source camera model of each input imagepatch) from the extracted noise-residual patterns. The system trains aneural network to learn the representation indicative of the statisticalfingerprint of the source camera model of each input image patch whileexcluding the semantic content thereof. The system determines a splicingmanipulation localization by the trained neural network.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing features of the invention will be apparent from thefollowing Detailed Description of the Invention, taken in connectionwith the accompanying drawings, in which:

FIG. 1 is a diagram illustrating the system of the present disclosure;

FIG. 2 is a flowchart illustrating overall processing steps carried outby the system of the present disclosure;

FIG. 3 is a flowchart illustrating step 54 of FIG. 2 in greater detail;

FIG. 4 is a flowchart illustrating step 72 of FIG. 3 in greater detail;

FIG. 5 is a table illustrating a neural network architecture of thesystem of the present disclosure;

FIG. 6 is a flowchart illustrating step 56 of FIG. 2 in greater detail;

FIGS. 7A-B are graphs illustrating a range of regularization parametervalues of an RD curve and a performance of selected regularizationparameter values;

FIGS. 8A-B are tables illustrating quantitative processing results ofthe system of the present disclosure on different datasets and accordingto different scoring metrics;

FIG. 9 is a compilation of images illustrating qualitative processingresults of the system of the present disclosure in comparison to avariety of models based on several input images and ground truth masksthereof;

FIG. 10 is a compilation of images illustrating qualitative processingresults of the system of the present disclosure based on a detection ofa variety of known generative adversarial network signatures present ininput images;

FIG. 11 is a diagram illustrating hardware and software componentscapable of being utilized to implement the system of the presentdisclosure; and

FIG. 12 is a diagram illustrating the neural network 16 of the presentdisclosure, in greater detail.

DETAILED DESCRIPTION

The present disclosure relates to systems and methods for improvedlocalization of image forgery, as described in detail below inconnection with FIGS. 1-12 .

By way of background, the image formation process broadly consists ofthree stages: (1) sensor measurements; (2) in-camera processing; and (3)storage which may include compression. The image formation process isunique for every camera model and yields subtle distortions and noisepatterns (i.e., a digital camera model fingerprint) in the image whichare invisible to the eye. These subtle distortions and noise patternsare useful in forensic applications because they are specific to eachcamera model. Sensor pattern noise originates from imperfections in thesensor itself and has shown to be sensitive to several manipulationtypes. Accordingly, sensor pattern noise has been utilized for thedetection and localization of forgeries. However, sensor pattern noiseis difficult to detect in image regions with high texture and is absentor suppressed in saturated and dark regions of an image. Color filterarray (CFA) demosaicking is an in-camera processing step that producespixel colors. Different detection and localization strategies based onCFA signature inconsistencies are known. However, the scope of suchspecialized CFA models is often limited. Joint Photographic ExpertsGroup (JPEG) is a common storage form and carries camera modelsignatures such as dimples or can contain clues regardingpost-processing steps such as traces of multiple compressions. Although,JPEG statistics have been utilized for detection and localization tasks,these statistics are format specific and do not generalize to othercommon or new formats.

More generic approaches include modeling noise-residuals. Anoise-residual is a statistical pattern that is not attached to aspecific source but is instead a result of the combined processes of animaging pipeline. Noise-residuals can be discerned by suppressing thesemantic contents of an image. For example, one known approach utilizesa wavelet transform as a high-pass filter to estimate noise-residualsand then determine its inconsistencies. Other known approaches utilizespatial rich filters (RFs). Spatial RFs are a set of alternate high-passfilters to model local noise-residuals. For example, one known approachexplores co-occurrences of one RF while another known approach utilizesthree residual filters along with color information in a convolutionalneural network (CNN) to localize manipulations. Learned RFs utilizingconstrained convolutions have also been employed for localizingmanipulations. Noiseprint, another known approach, utilizes a denoisingCNN to estimate properties of noise-residuals and changes therein todiscover manipulations. Additionally, another known approach, utilizes aCNN trained for camera model identification to discover manipulationsbut does not exploit noise-residuals and relies on a CNN architecture tolearn semantic contents.

The system of the present disclosure utilizes a constrained convolutionlayer to mimic RFs and information bottleneck to learn noise-residualpatterns and localize manipulations. The system of the presentdisclosure demonstrates that information bottleneck provides for aformal framework to interpret mutual information-based regularization.This interpretation provides for a more efficient solution utilizingvariational approximation and provides for tuning the regularization ina principled manner to enhance forensic performance.

Information theory is a framework that provides for improving variousaspects of deep machine learning including, but not limited to,representation learning, generalizability and regularization, and aninterpretation of deep neural network functionality. Mutual informationplays an important role in many of these methods. For example, InfoGANhas shown that maximizing mutual information between latent code and agenerator's output improves the representations learned by a GAN therebyproviding for the representations to be more disentangled andinterpretable. Mutual information is challenging to compute andtherefore InfoGAN maximizes a variational lower bound. A similarmaximization approach has been explored to improve unsupervisedrepresentation learning utilizing a numerical estimator.

Information bottleneck curbs information flow between an input and arepresentation layer. This curbing of information flow encourages amodel to learn task related features and helps improve itsgeneralization ability. It should be understood that informationbottleneck Lagrangian is challenging to solve in practice. Accordingly,variational approximations suitable for deep learning have been proposedand demonstrate that information bottleneck is closely related tovariational autoencoders (VAEs). Information bottleneck can be utilizedto learn disentangled representations. For example, informationbottleneck approaches to improve the disentanglement of representationslearned by VAEs have been investigated empirically. Additionally, aknown insightful rate-distortion interpretation utilizing informationbottleneck has been applied to VAEs. Information bottleneck has alsobeen proposed as an effective regularization and shown to improveimitation learning, reinforcement learning, and the training of GANs.The system of the present disclosure leverages a variational informationbottleneck formulation developed for deep neural networks utilizing areparameterization process.

The information bottleneck framework and its variational approximationwill now be described. Learning a predictive model p(y|x) is hinderedwhen a model overfits nuisance detractors that exist in the input data Xinstead of focusing on relevant information for a task Y. This is ofimportance in deep learning when an input is high-dimensional (e.g., animage), a task is a simple low-dimensional class label, and a model is aflexible neural network. An objective of information bottleneck is toovercome this problem by learning a compressed representation Z, of X,which is optimal for the task Y in terms of mutual information. It isapplied by maximizing the information bottleneck Lagrangian based onmutual information values I(Z, X) and I(Z, Y) as follows in Equation 1:

=I(Z,Y)−βI(Z,X)   Equation 1

By penalizing an information flow between X and Z while maximizing themutual information required for the task, information bottleneckextracts the relevant information that X contains regarding Y anddiscards non-informative signals. This provides for learning arepresentation Z with an improved generalization ability.

As mentioned above, it should be understood that mutual information ischallenging to compute in a general setting and even more so with highdimensional variables. As such, a known approach applies a variationalapproximation to a neural network. In particular, let Z be a stochasticencoding layer and based on the definition of mutual information,Equation 2 yields:

$\begin{matrix}{{I\left( {Z,Y} \right)} = {{\int{{p\left( {y,z} \right)}\log\frac{p\left( {y,z} \right)}{{p(y)}{p(z)}}{dydz}}} = {{\int{{p\left( {y,z} \right)}\log\;{p\left( y \middle| z \right)}{dydz}}} - {\int{{p(y)}\log\;{p(y)}{dy}}}}}} & {{Equation}\mspace{14mu} 2}\end{matrix}$

In Equation 2, the last term is ignored as it is the entropy of y and isconstant. The first term p(y|z) is intractable and is approximatedutilizing a variational distribution q(y|z), the decoder network. Then,a lower bound of I(Z, Y) is determined because the KL divergenceKL[p(y|z)∥q(y|z)]≥0⇒∫p(y|z)log p(y|z)dy≥∫p(y|z)log q(y|z)dy and byassuming a Markov chain relation Y→X→Z, yields Equation 3:I(Z,Y)≥

_(x,y˜p(x,y))

_(z˜p(z|x))[log q(y|z)]]   Equation 3

where p(z|x) is an encoder network and p(x, y) can be approximatedutilizing the training data distribution. Therefore, the right hand sideof Equation 3 becomes the average cross-entropy (with stochasticsampling over z). Proceeding similarly, Equation 4 yields:

$\begin{matrix}{{I\left( {Z,X} \right)} = {{\int{{p\left( {x,z} \right)}\log\frac{p\left( {x,z} \right)}{{p(x)}{p(z)}}{dxdz}}} = {{\int{{p\left( {x,z} \right)}\log\;{p\left( z \middle| x \right)}{dxdz}}} - {\int{{p(z)}\log\;{p(z)}{dz}}}}}} & {{Equation}\mspace{14mu} 4}\end{matrix}$

In this case, p(z) is intractable and is approximated by a priormarginal distribution r(z). An upper bound for I(Z, X) is determinedbecause KL[p(z)∥r(z)]≥0⇒∫p(z) log p(z)dz≥∫p(z)log r(z)dz, thereforeEquation 5 yields:

$\begin{matrix}{{{I\left( {Z,X} \right)} \leq {\int{{p(x)}{p\left( z \middle| x \right)}\log\frac{p\left( z \middle| x \right)}{r(z)}{dxdz}}}} = {{\mathbb{E}}_{x \sim {p{(x)}}}\left\lbrack {{KL}\left\lbrack {{p\left( z \middle| x \right)}{}{r(z)}} \right\rbrack} \right\rbrack}} & {{Equation}\mspace{14mu} 5}\end{matrix}$

p(x) can be approximated utilizing the data distribution. ReplacingEquations 3 and 5 in Equation 1 yields the variational informationbottleneck function as shown in Equation 6:

$\begin{matrix}{{J_{IB}\left( {p,q} \right)} = {{{\frac{1}{N}{\sum\limits_{i = 1}^{N}{{\mathbb{E}}_{z \sim {p{({z|x_{i}})}}}\left\lbrack {{- \log}\mspace{11mu}{q\left( y_{i} \middle| z \right)}} \right\rbrack}}} + {\beta\;{{KL}\left\lbrack {{p\left( z \middle| x_{i} \right)}{}{r(z)}} \right\rbrack}}} \geq {- \mathcal{L}}}} & {{Equation}\mspace{14mu} 6}\end{matrix}$

The variational information bottleneck function can be minimizedutilizing a known reparameterization process. According to arate-distortion interpretation of information bottleneck, the loss termis denoted as distortion D and approximates the non-constant part of−I(Z, Y) while the unweighted regularization term is denoted as rate Rand approximates I(Z, X). R measures an excess number of bits requiredto encode representations. The RD-plane provides for visualizing afamily of solutions to the information bottleneck Lagrangian fordifferent values of β and provides insight into properties of theencoder-decoder network.

The system of the present disclosure allows for localization of adigital manipulation that inserts foreign material into a host image toalter its contents (e.g., a splicing operation). Since a splicingoperation is often camouflaged by semantic structures, suchmanipulations can be localized by inspecting low-level pixel statistics.Generally, a splicing operation will contain a different statisticalprofile (e.g., fingerprint) than the host image because the splicingoperation likely originates from a different camera model or a differentimage formation process (e.g., inpainting). The system of the presentdisclosure allows for localization of an image manipulation by utilizingan information bottleneck-based loss to learn to ignore semantic contentof an image. In particular, the system trains a deep neural network tolearn a representation that captures a statistical fingerprint of asource camera model from an input image patch while ignoring thesemantic content of the input image patch. Then, the system computes thefingerprint representation for different parts of a test image. Lastly,the system searches for inconsistencies among the computed fingerprintrepresentations to localize splicing manipulations. It should beunderstood that the system trains the neural network with a large numberof camera models to improve the ability of the system to distinguisheven unseen camera models. Accordingly, the network can be effective ina blind test context when applied on images acquired from unknown cameramodels.

Turning to the drawings, FIG. 1 is a diagram illustrating the system 10of the present disclosure. The system 10 includes a neural network 16having an information bottleneck function generation module 14 thatreceives input data 12. The neural network 16 can receive training inputdata 20 and validation input data 24. The neural network 16 furtherincludes a model training system 18 and a trained model system 22, andoutputs output data 26. The neural network 16 can be any type of neuralnetwork or machine learning system, or combination thereof. For example,the neural network 16 can be a deep neural network capable of, forexample, image forgery localization and can use one or more frameworks(e.g., interfaces, libraries, tools, etc.).

FIG. 2 is a flowchart 50 illustrating overall processing steps carriedout by the system 10 of the present disclosure. Beginning in step 52,the information bottleneck function generation module 14 generates avariational information bottleneck objective function. The informationbottleneck function generation module 14 receives input data 12.Utilizing information bottleneck provides for framing the issue oflocalizing manipulations in an image as a deep representation learningproblem and provides a framework for interpreting a deep neural network.In particular, utilizing information bottleneck casts the problem ofmodelling and distinguishing low-level camera model statistics as adata-driven representation learning problem. The system 10 works withinput image patches to implement an encoder-decoder architecture. Theencoder-decoder architecture controls an information flow between theinput image patches and a representation layer. This constriction ofmutual information allows the neural network 16 to ignore irrelevantsemantic content contained in any image patch and focus its capacity onlearning useful features required to classify a source camera model. Assuch, the system 10 utilizes information bottleneck to learn usefulresidual noise patterns and ignore semantic content. As mentioned above,a learned noise pattern representation can be considered as a cameramodel signature (i.e., fingerprint).

In step 54, the model training system 18 trains the neural network 16utilizing the information bottleneck function on training input data 20.In particular, the model training system 18 trains the neural network 16to learn a representation indicative of a statistical fingerprint of asource camera model from an input image patch while excluding semanticcontent thereof. The training input data 20 can include, but is notlimited to, a predetermined number of images of the Dresden ImageDatabase that contains more than 17,000 JPEG images from 27 sourcecamera models. It should be understood that the neural network 16 can beany type of neural network or machine learning system, or combinationthereof, utilizing the information bottleneck function. Additionally, itshould be understood that the system 10 may utilize a different neuralnetwork 16 based on the training input data 20. For example, the system10 may utilize a model with mutual information regularization and amodel without mutual information regularization when training on theDresden Image Database. Then, in step 56, the trained model system 22processes validation input data 24 to determine whether the system 10can localize an image manipulation. The validation input data 24 caninclude, but is not limited to, a predetermined number of images of theDSO-1, Nimble Challenge 2016 (NC16) and the Nimble Challenge 2017(NC17-dev1) datasets.

FIG. 3 is a flowchart illustrating step 54 of FIG. 2 in greater detail.The system 10 learns a low level representation by extractingnoise-residuals and learning suitable representations from the extractednoise-residuals. Beginning in step 70, the system 10 extractsnoise-residuals by considering learned local noise models. Inparticular, the system 10 considers a constrained convolution layeraccording to Equation 7 as follows:

$\begin{matrix}{\mathcal{R}^{(k)} = {{{w_{k}\left( {0,0} \right)} + {\sum\limits_{i,{j \neq 0},0}{w_{k}\left( {i,j} \right)}}} = 0}} & {{Equation}\mspace{14mu} 7}\end{matrix}$

The constrained convolution layer binds the weights of the k^(th) filterto determine a mismatch or noise-residual, between a pixel's value atposition (0, 0) and its value as interpolated from its S×S neighbors.These are high-pass filters similar to RFs that model noise-residualslocally by suppressing semantic content and can be trained end-to-end byincluding a penalty

=(Σ_(k)(

^((k)))²)^(1/2) in the optimization.

It should be understood that since these noise models are high-passfilters, the models also capture high-frequency semantic content such asedges and textures which carry scene related information the system 10seeks to suppress. Ideal noise-residuals are considered to behigh-frequency content uncorrelated to semantic information. It ispossible to learn these noise-residuals by regularizing mutualinformation between an input and a feature layer in a neural network.Intuitively, this would discourage a correlation between learnedfeatures and semantic content in the input. However, and as mentionedabove, mutual information is challenging to compute. Accordingly, thesystem 10 re-interprets the mutual information regularization throughthe information bottleneck framework. As such, the system 10 can employan efficient variational solution and explore longer training processesand provide an RD-plane that can be evaluated to select the bestregularization parameter β of Equation 6. In step 72, the system 10learns suitable representations from the extracted noise-residuals. Step72 will be described in further detail below in connection with FIG. 4 .

FIG. 4 is a flowchart illustrating step 72 of FIG. 3 in greater detail.Beginning in step 90, the system 10 implements a stochasticencoder-decoder architecture to learn suitable representations utilizinginformation bottleneck. As the input, the system 10 considers X to be animage patch, Y to be a class label for a task of camera modelidentification and Z to be a stochastic encoding layer. The system 10selects a classification task to learn non-semantic features from theinput image patch since the semantic content of an image is not relevantto the classification of a camera model. Additionally, the system 10 canexploit large camera tagged untampered image databases for trainingwhich provides for avoiding specialized manipulated datasets andaverting the chances of overfitting to the specialized manipulateddatasets. As implemented, the system 10 can train the neural network 16by minimizing the variational IB objective function of Equation 6.

In step 92, the system 10 configures the encoder p(z|x). The system 10can utilize an architecture inspired by residual network 18 (ResNet-18)version-1 including an initial constrained convolution layer (as shownin Equation 7) to model noise-residuals and discard operations thatquickly shrink the input and encourage learning high level (i.e.,semantic) features. Namely, the system 10 discards an initialmax-pooling layer, convolutions with a stride greater than one, and afinal global average pooling layer. The system 10 also insertsadditional 7×7 and 5×5 convolutions to end the network with a single“feature-pixel” with a large bank of filters to avoid fully connectedlayers.

FIG. 5 is a table 100 illustrating the architecture of the neuralnetwork 16 of the system 10 of the present disclosure. In particular,the neural network 16 is a CNN having 27 layers where every convolutionis followed by batch normalization and ReLU activation. The convolutionshave a stride of 1. The input patch size is 49×49×3 and the outputencoding size is 1×1×72. To yield a stochastic encoding Z, the system 10splits the CNN's output vector of 72 filters into μ_(x) and σ_(x) andmodel p(z|x)=N(μ_(x), diag(σ_(x))). The neural network 16 also includesa decoder, discussed in greater detail below in connection with FIG. 12.

Returning to FIG. 4 , in step 94 the system 10 configures the decoderq(y|z). The decoder deters the CNN from degenerating to the autodecoderlimit, an issue faced by VAEs with powerful decoders. As such, thesystem 10 configures the decoder with a simple logistic regression modelhaving a dense (log it generating) layer that is connected to thestochastic code layer Z and is activated by the softmax function.

Then, in step 96, the system 10 determines the regularization parameterβ. The system 10 utilizes the RD-plane to determine the characteristicsof the encoder-decoder. It should be understood that an RD curve dividesthe plane into practical feasible and infeasible regions. Evaluating theRD curve provides for selecting a regularization parameter β to balancea trade-off between the distortion which affects task accuracy and therate which affects compression and hence the generalization capacity. Inaddition to a primary task of localizing splicing manipulations, thesystem 10 also provides for training the neural network 16 on asecondary task of camera model identification. As such, the system 10employs the RD curve of the training task to identify a potential rangefor the regularization parameter β and then selects optimal value(s) ofthe regularization parameter β from this range through empiricaltesting.

FIG. 6 is a flowchart illustrating step 56 of FIG. 2 in greater detail.The system 10 simplifies localizing a splicing manipulation into atwo-class feature segmentation problem based on the assumption that theuntampered region of a test image is the largest part of the test image.Beginning in step 120, the system 10 determines the neural network's 16representation (μ, σ) as a predictive signature of a camera model forjuxtaposed patches in the test image. Then, in step 122, the system 10segments the predictive signature via a Gaussian mixture model with twocomponents utilizing expectation maximization (EM). This segmentationstep 122 is described in greater detail below in connection with FIG. 12, and it is noted that the Gaussian mixture model is not a part of theneural network 16 described herein. It should be understood that theGaussian distributions are approximate statistics of the features of thetwo classes and aid in separating the classes probabilistically. Itshould also be understood that the system 10 does not perform forgerydetection since the system 10 identifies two classes.

Training and testing of the neural network 16 of the system 10 will nowbe described. The system 10 evaluates input patches of size 49×49×3 andk=64 constrained convolutions with support S=3. The encoder has a fixednumber of 64 filters in every layer. Additionally, for the variationalprior distribution, the system 10 utilizes a factorized standardGaussian r(z)=Π_(i)N_(i)(0, 1) and trains the neural network 16utilizing the loss of Equation 8 as follows:J=J _(IB)+λ

+ω₁ ∥W∥ ₁+ω₂ ∥W∥ ₂   Equation 8

In Equation 8, W denotes all weights of the neural network 16 and thesystem 10 empirically selects λ=1 and ω₁=ω₂=1e−4. As mentioned above,the system 10 can utilize the Dresden Image Database as the traininginput data 20. The Dresden Image Database consists of more than 17,000JPEG images corresponding to 27 source camera models. For each cameramodel, the system 10 randomly selects 70% of the images for training,20% for validation and 10% for testing. The system 10 trains with amini-batch of 200 patches for 700 epochs with 100,000 randomly selectedpatches in every epoch. The system 10 maintains a constant learning rateof 1e-4 for 100 epochs which then linearly decays to 5e-6 in the next530 epochs and finally exponentially decays by a factor 0.9 over thelast 70 epochs. As such, the system 10 provides for a camera modelprediction accuracy of ˜80% on the test and validation sets for variousvalues of the regularization parameter β.

The system 10 was implemented with TensorFlow and trained on a NVIDIATesla V100-SXM2 (16 GB) GPU with 32 CPUs and 240 GB RAM. It should beunderstood that the system 10 can be implemented utilizing othersoftware and hardware configurations. For comparison, a deep networkhaving 18 layers with 64 filters (instead of 19) was also trained. Thisnetwork trained with 72×72×3 input patches for the same number of epochsbut with a decreased batch size of 100. Additionally, the system 10trains two neural network models. In particular, the system 10 trains aneural network model with mutual information regularization (MI) and aneural network model without mutual information regularization (NoMI).Training the variational model of the system 10 required 14 hourswhereas training the MI model required eight days thereby highlightingthe efficiency of the variational solution in contrast to thenumerically expensive binning method (i.e., the MI model).

The processing results of the system 10 will now be described. Thesystem 10 can be tuned by evaluating the RD curve and selecting anoptimal regularization parameter β. It is noted that an ablation studyis carried out to gauge a relevance of information bottleneck. Thesystem 10 is tested on three standard manipulated datasets and scoresare generated via three distinct metrics. The manipulation datasetsinclude, but are not limited to, a predetermined number of images of theDSO-1, NC16, and the NC17-dev1 datasets. The DSO-1 dataset consists of100 spliced images in Portable Network Graphics (PNG) format such thatthe tampered regions are relatively large but well camouflaged by thesemantic content of each image. The NC16 dataset consists of 564 splicedimages mostly in JPEG format. The NC17-dev-1 dataset consists of 1,191images having different types of manipulations. Of these images, onlyspliced images are selected thereby yielding 237 images. The NC16 andNC17-dev-1 images contain a series of manipulations, some of which arecomplex operations that attempt to erase traces of manipulations.Furthermore, the tampered regions are often small. Each of the DSO-1,NC16 and NC17-dev1 datasets contain difficult to detect manipulationsand are accompanied by ground truth manipulation masks. Additionally,manipulations created by three well known inpainting GANs are generated.

Performance of the system 10 was evaluated via three metrics includingthe F1 score, the Matthews Correlation Coefficient (MCC) and an areaunder the receiver operating characteristic curve (ROC-AUC). Thesemetrics are known for evaluating a splicing manipulation localization.F1 and MCC require a binarized forgery prediction mask while the system10 predicts probabilities from the EM segmentation. It is customary togenerate and report scores for optimal thresholds computed from theground truth masks. As such, scores from automatic thresholdingutilizing a known method (e.g., Otsu's method) are generated andreported.

For comparison with the system 10, two ablated models are consideredincluding the neural network model with mutual informationregularization (MI) and the neural network model without mutualinformation regularization (NoMI) as described earlier. An optimalregularization parameter β is selected in addition to a variationalmodel with no information bottleneck regularization (β=0). These modelsand regularization values aid in gauging the importance of informationregularization and provide a comparison of the efficient variationalapproach of the system 10 and the expensive numerical binning approach(i.e., the MI model). Additionally, other models are consideredincluding the SpliceBuster (SB) which is a state-of-the-art splicelocalization algorithm and top performer of the NC17 challenge and theEX-SC which is a deep learning based algorithm that predicts meta dataself-consistency to localize tampered regions.

FIGS. 7A-B are graphs illustrating a range of regularization parametervalues of an RD curve and a performance of selected regularizationparameter values. In particular, FIG. 7A is a graph 140 illustrating aplot of an RD curve 142 from which regularization parameter values β canbe selected. As shown in FIG. 7A, the system 10 yields low distortionvalues for β≤5e-3 for the training task. FIG. 7B is a graph 150illustrating F1 scores on the DSO-1 dataset. In selecting aregularization parameter β for the forensic task, F1 scores aredetermined on the DSO-1 dataset for all values of β until 0. As shown inFIG. 7B, a peak in the F1 scores is evident from 2e-3 to 1e-4 (1e−3 isan anomaly that can be attributed to stochastic training). As such,testing is carried out for the central regularization parameter valuesof β=1e−3 and 5e−4.

FIGS. 8A-8B are tables illustrating quantitative processing results ofthe system 10 on different datasets and according to different scoringmetrics. In particular, FIG. 8A is a table 160 illustrating quantitativeresults for splicing manipulation localization of the system 10 andother models on the DSO-1, NC16, and NC17-dev1 datasets according to theF1 and MCC scoring metrics. Scores listed in each of the left columnsare indicative of optimal thresholds and scores listed in each of theright columns are indicative of automatic thresholding utilizing a knownapproach (e.g., Otsu's method). As shown in FIG. 8A, the F1 scoresindicate up to a 6% point improvement over SB and a 15% pointimprovement over EX-SC on the DSO-1 dataset and the best scores on theNC16 and NC17-dev1 datasets. The system 10 IP1e-3 and IP5e-4 model MCCscores are also high in comparison to the other models with a margin ofup to 8% points on the DSO-1 dataset in comparison to SB. FIG. 8B is atable 170 illustrating quantitative results for splicing manipulationlocalization of the system 10 and other models on the DSO-1, NC16, andNC17-dev1 datasets according to the AUC scoring metric. As shown in FIG.8B, a comparison of the ablated models NoMI and MI evidences thatinformation regularization improves forensic performance. Additionally,FIG. 8B illustrates that the variational approach of the system 10(e.g., the IP1e-3 and IP5e-4 models) outperforms the numericallyexpensive MI model.

FIG. 9 is a diagram 180 of a compilation of images illustratingqualitative processing results of the system 10 in comparison to the MI,EX-SC and SB models based on several input images 182 a-f and groundtruth masks 184 a-f thereof. The input images 182 a-f are sourced fromthe DSO-1, NC16 and NC17-dev1 datasets. As shown in FIG. 9 , the system10 images 186 a-f closely resemble the ground truth images 184 a-f ofthe input images 182 a-f. Further, the system 10 images 186 a-f moreclosely resemble the ground truth images 184 a-f than the MI modelimages 188 a-f, the EX-SC model images 190 a-f and the SB model imagesa-f.

FIG. 10 is a compilation of images 200 illustrating qualitativeprocessing results of the system 10 based on a detection of a variety ofknown GAN signatures present in input images 210 a-f. As shown in FIG.10 , the system 10 detects the signatures of the inpainting GANs in theinput images 210 a-f. It should be understood that most of the inputimages 210 a-f are processed (e.g., resized or compressed) whichdestroys camera model traces but nevertheless the system 10 is able todetect the splicing manipulations present in the input images 210 a-f.Accordingly, this indicates that synthetically generated pixels carry adifferent low level signature that the system 10 can detect and exploit.

The system 10 utilizes an information bottleneck formulation thatconverts a classical feature modelling problem for identifying cameramodels into a deep representation learning problem. This is a uniqueapplication of information bottleneck to a growing real-world problemwith serious consequences. The application of information bottleneck viathe system 10 is also unique in that it encourages learning low levelnoise patterns rather than semantic information which is contrary to theconventional application of information bottleneck. A comparison of thesystem 10 with the expensive number estimation method (i.e., the MImodel) evidences that the computationally efficient approximatedsolution based on variational information bottleneck of the system 10outperforms the MI model. As such, the representation learning problemcan be solved numerically or approximated via the variational inferencewhere the latter outperforms the former in regards to the task ofsplicing manipulation localization. Additionally, the system 10outperforms several state of the art models on a suite of standard testdatasets and can detect the signatures of deep generative models (e.g.,inpainting GANs).

FIG. 11 is a diagram 300 showing hardware and software components of acomputer system 302 (i.e. a server) on which the system 10 of thepresent disclosure can be implemented. The computer system 302 caninclude a storage device 304, computer software code 306, a networkinterface 308, a communications bus 310, a central processing unit (CPU)(microprocessor) 312, a random access memory (RAM) 314, and one or moreinput devices 316, such as a keyboard, mouse, etc. It is noted that theCPU 312 could be one or more graphics processing units (GPUs), ifdesired. The server 302 could also include a display (e.g., liquidcrystal display (LCD), cathode ray tube (CRT), etc.). The storage device304 could comprise any suitable, computer-readable storage medium suchas disk, non-volatile memory (e.g., read-only memory (ROM), erasableprogrammable ROM (EPROM), electrically-erasable programmable ROM(EEPROM), flash memory, field-programmable gate array (FPGA), etc.). Thecomputer system 302 could be a networked computer system, a personalcomputer, a server, a smart phone, tablet computer etc. It is noted thatthe server 302 need not be a networked server, and indeed, could be astand-alone computer system.

The functionality provided by the present disclosure could be providedby computer software code 306, which could be embodied ascomputer-readable program code stored on the storage device 304 andexecuted by the CPU 212 using any suitable, high or low level computinglanguage, such as Python, Java, C, C++, C#, .NET, MATLAB, etc. Thenetwork interface 308 could include an Ethernet network interfacedevice, a wireless network interface device, or any other suitabledevice which permits the server 302 to communicate via the network. TheCPU 312 could include any suitable single-core or multiple-coremicroprocessor of any suitable architecture that is capable ofimplementing and running the computer software code 306 (e.g., Intelprocessor). The random access memory 314 could include any suitable,high-speed, random access memory typical of most modern computers, suchas dynamic RAM (DRAM), etc.

FIG. 12 is a diagram illustrating the neural network 16 of FIG. 1 ingreater detail. As can be seen, the network takes an input patch 350from an input image 360, during training of the neural network. Theinput patch 350 is processed by an encoder 354 having one or more richfilters, which generate semantic edges as can be seen in the image 364.A feature extractor 356 extracts a noise fingerprint using the processesdisclosed herein, to produce a camera noise fingerprint shown in image366. The noise fingerprint is processed by a decoder 358 and associatedclassifier to perform a proxy task of camera classification. Also, thenoise fingerprint is processed by a Gaussian mixture model 360 toperform a main task of splice localization, as illustrated in image 368.The Gaussian mixture model 360 is not a part of the network 16, and isinstead a separate model utilized for splice localization. The splicelocalization shown in image 368 identifies the genuine patches of theimage from the forged ones, thereby indicating the forged portions ofthe image. As can be seen in FIG. 12 , the network 16 utilizes theinformation bottleneck function described herein during the filtration,encoding, feature extraction, sampling, decoding, and classificationsteps. The information bottleneck function improves over rich filters bysuppressing the semantic contents and amplifying the true camera noisefingerprints in the image, thereby enhancing the ability of the systemto determined forged portions of an image.

Having thus described the system and method in detail, it is to beunderstood that the foregoing description is not intended to limit thespirit or scope thereof. It will be understood that the embodiments ofthe present disclosure described herein are merely exemplary and that aperson skilled in the art can make any variations and modificationwithout departing from the spirit and scope of the disclosure. All suchvariations and modifications, including those discussed above, areintended to be included within the scope of the disclosure.

What is claimed:
 1. A machine learning system for localizing imageforgery comprising: a memory; and a processor in communication with thememory, the processor: generating a variational information bottleneckfunction, training a neural network with the variational informationbottleneck function and a plurality of images of a dataset to learn alow level representation indicative of a statistical signature of asource camera model of each image among the plurality of images, andlocalizing a forgery within an image of the dataset using the trainedneural network, wherein the processor: extracts at least one noiseresidual pattern from each image among the plurality of images,constructs the neural network based on the extracted at least one noiseresidual pattern to suppress semantic information from each image amongthe plurality of images, and trains the neural network by minimizing thevariational information bottleneck function.
 2. The system of claim 1,wherein the processor, determines a range of values of a regularizationparameter of the variational information bottleneck function based on anRD curve, and determines a value of the regularization parameter basedon the range.
 3. The system of claim 1, wherein the processor localizesthe attribute of the image of the dataset by the trained neural networkby: determining a low level representation for juxtaposed patches of theimage based on the learned low level representation indicative of thestatistical signature of the source camera model of the image, andsegmenting the determined low level representations of the juxtaposedpatches.
 4. The system of claim 1, wherein the neural network is a 27layer encoder-decoder based Convolutional Neural Network (CNN) whereineach convolution is followed by batch normalization and a rectifiedlinear unit (ReLU) activation.
 5. The system of claim 1, wherein thedataset is a Dresden Image dataset.
 6. The system of claim 1, whereinthe localized adversarial perturbation of the image is at least one of asplicing or an inpainting manipulation.
 7. A method for localizing imageforgery by a machine learning system, comprising the steps of:generating a variational information bottleneck function, training aneural network with the variational information bottleneck function anda plurality of images of a dataset to learn a low level representationindicative of a statistical signature of a source camera model of eachimage among the plurality of images, localizing a forgery within animage of the dataset using the trained neural network, extracting atleast one noise residual pattern from each image among the plurality ofimages, constructing the neural network based on the extracted at leastone noise residual pattern to suppress semantic information from eachimage among the plurality of images, and training the neural network byminimizing the variational information bottleneck function.
 8. Themethod of claim 7, further comprising determining a range of values of aregularization parameter of the variational information bottleneckfunction based on an RD curve, and determining a value of theregularization parameter based on the range.
 9. The method of claim 7,further comprising localizing the attribute of the image of the datasetby the trained neural network by; determining a low level representationfor juxtaposed patches of the image based on the learned low levelrepresentation indicative of the statistical signature of the sourcecamera model of the image, and segmenting the determined low levelrepresentations of the juxtaposed patches.
 10. The method of claim 7,wherein the neural network is a 27 layer encoder-decoder basedConvolutional Neural Network (CNN) wherein each convolution is followedby batch normalization and a rectified linear unit (ReLU) activation.11. The method of claim 7, wherein the localized adversarialperturbation of the image is at least one of a splicing or inpaintingmanipulation.
 12. A non-transitory computer readable medium havinginstructions stored thereon for localizing image forgery by a machinelearning system which, when executed by a processor, causes theprocessor to carry out the steps of: generating a variationalinformation bottleneck function, training a neural network with thevariational information bottleneck function and a plurality of images ofa dataset to learn a low level representation indicative of astatistical signature of a source camera model of each image among theplurality of images, localizing a forgery within an image of the datasetusing the trained neural network, extracting at least one noise residualpattern from each image among the plurality of images, constructing theneural network based on the extracted at least one noise residualpattern to suppress semantic information from each image among theplurality of images, and training the neural network by minimizing thevariational information bottleneck function.
 13. The non-transitorycomputer readable medium of claim 12, the processor further carrying outthe steps of: determining a range of values of a regularizationparameter of the variational information bottleneck function based on anRD curve, and determining a value of the regularization parameter basedon the range.
 14. The non-transitory computer readable medium of claim12, the processor localizing the attribute of the image of the datasetby the trained neural network by carrying out the steps of: determininga low level representation for juxtaposed patches of the image based onthe learned low level representation indicative of the statisticalsignature of the source camera model of the image, and segmenting thedetermined low level representations of the juxtaposed patches.
 15. Thenon-transitory computer readable medium of claim 12, wherein the neuralnetwork is a 27 layer encoder-decoder based Convolutional Neural Network(CNN) wherein each convolution is followed by batch normalization and arectified linear unit (ReLU) activation.
 16. The non-transitory computerreadable medium of claim 12, wherein the localized adversarialperturbation of the image is at least one of a splicing or an inpaintingmanipulation.
 17. A machine learning system for localizing image forgerycomprising: a memory; and a processor in communication with the memory,the processor: generating a variational information bottleneck function,training a neural network with the variational information bottleneckfunction and a plurality of images of a dataset to learn a low levelrepresentation indicative of a statistical signature of a source cameramodel of each image among the plurality of images, and localizing aforgery within an image of the dataset using the trained neural network,wherein the processor: localizes the attribute of the image of thedataset by the trained neural network by: determining a low levelrepresentation for juxtaposed patches of the image based on the learnedlow level representation indicative of the statistical signature of thesource camera model of the image, and segmenting the determined lowlevel representations of the juxtaposed patches.
 18. The system of claim17, wherein the dataset is a Dresden Image dataset.
 19. The system ofclaim 17, wherein the localized adversarial perturbation of the image isat least one of a splicing or an inpainting manipulation.
 20. A methodfor localizing image forgery by a machine learning system, comprisingthe steps of: generating a variational information bottleneck function,training a neural network with the variational information bottleneckfunction and a plurality of images of a dataset to learn a low levelrepresentation indicative of a statistical signature of a source cameramodel of each image among the plurality of images, localizing a forgerywithin an image of the dataset using the trained neural network,determining a low level representation for juxtaposed patches of theimage based on the learned low level representation indicative of thestatistical signature of the source camera model of the image, andsegmenting the determined low level representations of the juxtaposedpatches.
 21. The method of claim 20, wherein the neural network is a 27layer encoder-decoder based Convolutional Neural Network (CNN) whereineach convolution is followed by batch normalization and a rectifiedlinear unit (ReLU) activation.
 22. The method of claim 20, wherein thelocalized adversarial perturbation of the image is at least one of asplicing or inpainting manipulation.
 23. A non-transitory computerreadable medium having instructions stored thereon for localizing imageforgery by a machine learning system which, when executed by aprocessor, causes the processor to carry out the steps of: generating avariational information bottleneck function, training a neural networkwith the variational information bottleneck function and a plurality ofimages of a dataset to learn a low level representation indicative of astatistical signature of a source camera model of each image among theplurality of images, localizing a forgery within an image of the datasetusing the trained neural network, determining a low level representationfor juxtaposed patches of the image based on the learned low levelrepresentation indicative of the statistical signature of the sourcecamera model of the image, and segmenting the determined low levelrepresentations of the juxtaposed patches.
 24. The non-transitorycomputer readable medium of claim 23, wherein the neural network is a 27layer encoder-decoder based Convolutional Neural Network (CNN) whereineach convolution is followed by batch normalization and a rectifiedlinear unit (ReLU) activation.
 25. The non-transitory computer readablemedium of claim 23, wherein the localized adversarial perturbation ofthe image is at least one of a splicing or an inpainting manipulation.